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Abstract 

.^The  bootstrap  is  proposed  as  a  method  for  estimating  the  precision  of 
forecasts  and  estimates  of  parameters  of  the  Kalman  Filter  model.  It  Is 
shown  chat  when  Che  system  and  the  filter  Is  In  steady  state  the  bootstrap 
applied  to  the  Gaussian  innovations  yields  asymptotically  consistent  standard 
errors.  That  the  bootstrap  works  well  with  moderate  sample  sizes  and  supplies 
robustness  against  departures  from  normality  Is  substantiated  by  emperlcal  ev¬ 
idence. 
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1.  Introduction 


The  Kalman  Filter  (KF)  has  become  an  Important  and  powerful  tool  for  the 
statistician.  Recently,  many  authors  have  exploited  the  state-space  model  and 
KF  recursions  for  estimation  and  prediction  of  time  series.  For  example, 

Jones  (1980)  and  Harvey  and  Pierse  (1984)  use  the  KF  to  obtain  maximum  likeli¬ 
hood  estimates  of  the  parameters  of  ARMA  processes  when  observations  are  miss¬ 
ing.  It  has  been  suggested  by  Morrison  and  Pike  (1977)  and  others  (cf.  Kendall 
(1973))  that  the  KF  model  provides  an  appropriate  setting  within  which  to 
parametrize  smoothing  and  forecasting  problems. 

To  be  specific,  we  suppose  that  a  p’^l  vector  time  series  (y^J  t“0,+l,+2. 
Is  being  generated  by  the  following  dynamic  system 

where  x^.  is  a  zero  mean,  pxl  vector  stationary  stochastic  signal,  and  v^  is 
pxi  Gaussian  white  noise,  v^~N(0,R).  The  dynamics  of  the  stationary  signal 
is  given  by 

where  $  is  the  pxp  transition  matrix  and  w^  is  p^l  Gaussian  white  noise, 
w^~N(0,Q).  Furthermore,  {v^}  and  {w^}  are  mutually  independent  and  we  assume 
that  the  system  and  the  filter  have  reached  steady  state.  We  remark  that  the 
superficially  more  general  model  in  vdiich  (1.1)  is  replaced  by 

Yt  ’MXt  +Vt 

where  M  is  a  nonsingular  known  design  matrix  may  be  reduced  to  (1.1)  by  an 
appropriate  change  of  bases. 
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Given  the  parameters  of  the  model,  namely,  $,  Q  and  R,  one  may  obtain  the 
minimum  mean  square  error  filter  and  forecasts  for  the  system  via  the  KF  re¬ 
cursions.  However  the  parameters  are  rarely  known  and  hence  must  be  estimated. 

It  is  clear  that  an  Inexact  filter  model  will  degrade  the  filter  perfor¬ 
mance.  In  fact,  such  an  inexact  model  may  cause  the  filter  to  diverge  (cf. 
Jazwlnskl  (1970),  pp.  244-251).  Hence,  the  precision  of  the  parameter  esti¬ 
mates  must  be  evaluated.  We  propose  the  bootstrap  as  a  method  to  evaluate  the 
precision  of  parameter  estimates,  in  particular,  to  handle  heteroscedastlclty , 
to  provide  robustness  against  departure  from  normality  in  the  Gaussian  state 
and  observation  errors,  and  to  assist  in  estimating  forecast  errors. 

Computationally  simple  estimates  of  the  parameters  of  the  KF  model  have 
been  given  by  Anderson,  Kleindorfer,  Kleindorfer,  and  Uoodroofe  (1969)  which 
we  henceforth  denote  by  AKKW.  Their  estimates,  which  are  discussed  in  Section 
4,  are  based  on  standard  ergodlc  theory  and  yield  strongly  consistent  estimates  of 
Q  and  R  under  minor  restrictions.  Hence  these  estimates,  which  do  not  assume 
a  Gaussian  likelihood,  have  none  of  the  drawbacks  of  the  iterative  maximum 
likelihood  techniques  such  as  Newton-Raphson  or  scoring  (cf.  Gupta  and  Mehra 
(1974))  or  the  EM  algorithm  (cf.  Shunway  and  Stoffer  (1981)  and  Wu  (1983))  which 
may  not  converge  or  converge  to  the  wrong  stationary  point. 

We  suggest  that  for  the  KF  model,  the  AKKW  estimates  are  the  roost  reason¬ 
able  to  bootstrap  since  the  theory  driving  the  estimators  is  sound.  We  make 
bootstrapping  these  estimates  appealing  by  showing  in  Section  5,  that  the 
bootstrap  gives  the  right  answers  with  large  samples,  that  is,  it  is  at  least 
as  sound  as  the  conventional  asymptotics.  Moreover,  if  the  investigator  of 
such  a  system  would  rather  rely  on  a  maximum  likelihood  iterative  scheme  for 
parameter  estimation,  the  AKKW  estimates  could  be  used  to  initialize  such 
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Iterative  procedures  since,  as  will  be  seen,  the  bootstrap  will  work  for  the 
initial  estimates. 

Finally,  in  Section  6,  we  give  emperlcal  evidence  of  the  bootstrap's  im¬ 
portance  in  Kalman  filtering  by  comparing  the  bootstrap  to  the  conventional 
asymptotics  in  the  cases  when  the  likelihood  is  Gaussian  and  when  the  likeli¬ 
hood  is  contaminated  Gaussian.  An  example  from  hydrology  is  also  given. 

Our  goal  is  to  estimate  the  precision  of  the  parameter  estimates  of  Q 
and  R,  as  well  as  the  precision  of  the  forecasts  ^n+l*^n+2 * "  ’ ’*n+k‘ 
techniques  used  here  are  based  on  the  bootstrap  (cf.  Efron  (1979))  and  the  methods 
used  in  bootstrapping  least  squares  estimates  discussed  in  Bickel  and  Freedman 
(1981),  Freedman  (1981)  and  Freedman  and  Peters  (1984).  It  is  noted  in  the 
above  references  that  in  regression  models  (static  or  dynamic) ,  it  is  appropriate 
to  resample  the  centered  residuals  after  estimating  the  parameters.  This  is 
not  possible  in  the  present  model  (1.1)  and  (1.2)  since  the  signal  is  not 
observable.  However,  we  may  base  the  procedure  on  the  Innovations  which  are 
obtained  by  taking  the  conditional  expectation  of  the  signal  given  the  data. 

Hence,  the  bootstrap  procedure  will  involve  the  resampling  of  the  innovation 
sequence 


t-1  t-1 

’  t*l,...,n 


(1.3) 


where  by  x^  ^  we  mean  E(x^  [y^^, . . .  ,y^_j^) .  Of  course  x^  ^  will  be  obtained  re¬ 
cursively  via  the  KF. 

Under  the  conditions  stated  in  the  next  section  we  will  be  able  to  put  this 
problem  into  the  nonlinear  regression  context  as  discussed  in  Efron  (1979, 
Section  7).  That  is,  we  may  write 


where  are  iid  zero  mean  random  vectors  (namely,  the  innovations)  and  g^(*)  is 
a  particularly  complicated,  but  known,  nonlinear  function  of  the  parameters 

$,  Q,  and  R,  the  signal  x^,  and  the  data  y^^ . ^t-l'  particular,  gj.(*)  = 

the  filtered  value  of  the  signal. 

In  the  next  section  we  give  conditions  under  i^lch  we  are  able  to  boot¬ 
strap  the  Innovations,  (1.3).  The  bootstrap  procedure  is  given  in  Section  3. 

2.  The  Steady-State  Innovation  Sequence 

Throughout  the  remainder  of  this  paper  we  make  the  following  assumptions 
on  the  pxp  parameter  matrices:  (Al)  Q  and  R  are  positive  definite,  and 
(A2)  is  nonsingular  with  spectal  norm,  p(<l>),  less  than  unity.  These  condi¬ 
tions  ensure  the  asymptotic  global  stability  of  the  KF  (cf.  Deyst  and  Price 
(1968)). 

The  steady-state  KF  recursions  are  given  by  (cf.  Jazwinskl  (1970)) 


K  =  P(P+R)”^, 

(2.1a) 

p  =  $[P_P(P+R)"^P]$'  +Q, 

(2.1b) 

t  A  t 

^t+1  '  “^^t  * 

(2.1c) 

x^  -  x^“^ +K(yj.  -  x^  ^). 

(2. Id) 

In  the  KF  above,  K  is  the  steady-state  gain  matrix,  P  is  the  steady-state 
filter  error,  P  =  E{  (x^-x^  ^)(x^-x^  ^)'},  and  x^”^  •  E(x^  ly^^, . . .  ,y^_^)  is  the 
steady-state  filter  estimate  of  x^  based  on  the  data 

Proposition  2.1  Under  steady-state  and  optimal  filtering,  the  pxl  vector  inno¬ 


vation  sequence 
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=  y  ~  X 
■'t  t 


(2.2) 


is  a  zero-mean,  white  Gaussian  sequence  with  covariance  matrix  P+R. 

Proof  Write  ®t**t~^t  ^ 

The  r^~^  are  Gaussian  since  they  are  linear  combinations  of  Gaussian  random  vec¬ 
tors.  To  establish  the  orthogonality  of  the  innovations,  it  is  easy  to  see  that 
while  r^~^(=  Vt  “  linear  space  spanned  by  (y^, . . .  .y^.} , 

r^“^(=  e^+v^)  is  orthogonal  to  the  linear  space  spanned  by  {y^^, . . .  ,y^_j^}. 

Hence,  for  s  <  t. 


Cr^'^r^"^')  =E{r®"^E(r^“^' jy^,...,yg  )>=0. 


Also,  since  and  are  uncorrelated  we  have  that 


Cov(r  )  *Cov(e  )  +Cov(v  )  ==P+R. 
c  t  t 


As  a  final  remark,  we  note  that  via  (2.1c)  and  (2. Id),  we  may  write  y^ 


in  terms  of  the  steady-state  innovations  as 


(2.3) 


which  follows  from  the  fact  that  II  11*0  exponentially  fast  as  j  -*•“  since 
2 

p(<t)<l,  where  ||4>||  =  trace($'$).  This  result  will  be  useful  in  establish¬ 

ing  the  bootstrap  procedure. 


3.  The  Bootstrap  Estimate  of  Precision 

As  previously  mentioned,  the  bootstrap  technique  will  be  employed  by  re- 
san5)ling  the  steady-state  innovation  sequence.  Recall  that  under  optimal 
filtering  the  innovation  sequence  r^  t=l,...,n  is  p^l  Gaussian  white  noise. 


^  ~  N(0,P+R)  where  P  Is  the  steady-state  error  covariance  matrix  given  in 
(2.1b). 


The  bootstrap  procedure  begins  be  estimating  the  parameters  0={$,Q,R}  of 
the  model  (1.1),  (1.2)  by  some  optimal  procedure  as  mentioned  in  the  Intro¬ 
duction.  We  shall  discuss  a  particular  method  in  Section  4.  Call  these  esti- 

A,  /s  y\ 

mates  0  =  {$,Q,R}. 

From  these  preliminary  estimates  obtain  a  suboptlmal  innovation  sequence 

^  ^t— 1 

by  filtering  (cf.  2.1)  under  0.  Call  this  innovation  sequence  r^  .  Make  the 
^  t  1  n 

sequence  {r^  ^t=l  identically  distributed  with  distribution 

equal  to  the  emperical  distribution  by  putting  mass  n  ^  on  each  innovation 
1 

r^  ,  t  =  l,...,n. 

*t-l 

Next,  draw  a  "bootstrap  sample"  of  innovations,  r^.  ,  t“l,...,n  by 

^t-1  ^t-1 

independent  random  sampling  of  the  residuals  r^  .  That  is,  sample  the  r^  ,n 

times,  with  replacement  from  {r^,r2, • . . }.  From  this  we  obtain  a  "boot- 

..  *  *  , 

strap  sample  of  data  ^y  setting  (cf.  2,3) 


*  *t-l  rt-1  ''  ''*t-i-l 


(3.1) 


where  K  is  the  estimated  gain  matrix  obtained  via  filtering  under  parameters 

0. 

We  make  the  following  suggestions  before  proceeding  with  step  (3.1). 

''t-1 

First,  as  suggested  in  Freedman  (1981),  one  should  center  the  residuals  r^ 
before  resampling  them  so  that  the  emperical  distribution  puts  mass  n  ^  on 
^t  "  ^t-l'^t  '  Second,  we  suggest  checking  whether  the  Innova 

tions  are  white.  It  is  known  that  a  suboptimal  filter  produces  correlated  in- 


novations  (see,  for  example,  Mehra  (1970))  and  hence  this  is  a  check  on  the 


"goodness"  of  the  estimates.  Various  methods  are  available  for  testing  the 


whiteness  of  the  innovations  many  of  which  are  listed  in  Mehra  (1970). 

*  * 


Now,  suppose  that  the  bootstrap  data  (y^^.  •  • .  come  from  the  model 

(3.2a) 
(3.2b) 


•k  k 

yt  =  Xt+Vt,  t>i. 


•k  k 

Xt  =  'I> 


where  is  pxl  Gaussian  white  noise  v^^N(0,R  )  and  is  independent  of  w^  which 

k  k  k  ^  k  k  k 

is  p^l  Gaussian  white  noise  w^  N(0,Q  ).  Assume  the  parameters  0  =  ,Q  ,R  } 


are  unknown  and  to  be  estimated. 


The  parameters  Q  are  then  estimated  by  the  initial  optimal  procedure  to 


produce  estimates  0  =  {$  ,Q  ,R  }.  Then,  the  suboptimal  innovation  sequence 


is  resampled  and  the  bootstrap  procedure  is  reiterated. 


The  entire  process  is  repeated  some  large  number  "L"  of  times  obtaining 
L  bootstrap  replications  0j^,02>*  •  •  The  distribution  of  the  errors 


$  Q  -Q,  R  -R 


(3.3) 


are  then  computed  to  give  an  approximation  as  to  the  distribution  of 

/N  /S  /V 

$  -  <!>,  Q-Q,  R-R.  (3.4) 


The  bootstrap  distribution  of  the  errors  (3.3)  may  then  be  used  to  obtain  con¬ 
fidence  regions  and  tests  of  hypotheses  about  the  parameters  0.  Justification 
of  this  procedure  is  given  in  Section  5,  Theorem  5.1. 

Forecasting  k  steps  into  the  future,  say  x”^^  =  E(x^^j  ly^^ , . . .  ,y^) , 
j*l,2,...,k  is  easily  accomplished  via  the  filter  equations  (2.1),  namely 
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Vj“V  -l-i.----' 


C3.5) 


The  suboptlmal  forecasts  will  be  obtained  via  the  KF  under  parameter  estimates 


Q  so  that 


n  *1  n  •  -1  1 

Vj  °  "'n* 


C3.6) 


will  be  the  actual  forecasts.  If  at  each  bootstrap  replication  we  obtain 


,C1) 


n*  ,a) 


f  u"  u-'  •>  f  «  «  1 

^^n+l’  *  • '  ’\+k^  *  •  •  •  ’  ^’‘n+1’  ‘  * '  ’*n+k^ 


we  may  extract  the  emperlcal  distribution  of  the  forecast  residuals 


*n+j"Vj’  3 I' 


C3.7) 


C3.8) 


which  can  then  be  used  to  approximate  the  distribution  of  the  actual  forecast 


n  n  ill 

Vj'Vj’  . 


C3.9) 


From  the  distributions  of  C3.8)  we  may  obtain  prediction  regions  for  the  forecasts 


C3.5). 


Consistent  Parameter  Estimates 


In  this  section  we  give  the  details  of  the  consistent  estimation  of  the  para 
meters  of  the  KF  model  U-.l),  Cl. 2).  Recall  that  the  system  is  in  steady-state 
and  that  the  parameters  0=  {t.Q.R}  satisfy  the  conditions  (Al)  and  (Ji2)  given  in 
Section  2. 

The  observation  sequence  may  then  be  written  as 


y  “V  +y.  n'f^W 

^t  t  t-j 


C4.1) 


from  which  it  follows  that  y  is  a  strictly  stationary,  zero-mean  Gaussian  se- 


9 


r  j  j 

quence  with  covariance  matrix  R  + 

The  estimates  of  $,  Q  and  R  are  based  on  the  matrices 

-1 


C4.2) 


which  in  view  of  C^.l)  and  the  Ergodic  Theorem  are  ergodic  and  converge  almost 
surely  Ca.s.)  to 

CD  I 

rCh)  = 

where  6,  „  is  the  Kronecker  6  and  FCh)  =  CovCy  *y  v,)* 
n,U  t  t“u 

An  estimate  of  $  is  suggested  by  the  fact  that  ECy^y^_2)  “ 
namely, 


$ 

n 


(4.4) 


where  by  +  we  mean  generalized  inverse.  Estimates  of  Q  and  R  are  suggested 
by  the  facts  that 


B(l)  =  E{(y^-<lyj._j^)(y|.-4.y|._^)'} 

=  Q  +  R  +  <I>R<I'' 

B(2)  =E{(y|.-$Sj._2)(y^-<I>^yt_2)'} 

2  2' 

=  Q  +  R  +  +$  R<I> 

which  yield 

R  =  |{B(1)  +«>"^(B(1)  -B(2))<J>~^'}  (4.5) 


Q  =  B(l)  -  R  -  4>R4>' 


(4.6) 


provided  that  is  nonsingular. 

We  now  state  the  following  theorems  which  follow  directly  from  the  almost 
sure  convergence  of  (4.2)  to  (4.3).  Denote  the  spectral  norm  of  $  by  p(1>). 

Theorem  4.1  If  p('J>)  <1.  and  if  <I>  and  Q  are  nonsingular,  then  given  in  (4.4) 
is  strongly  consistent  for  $. 


Theorem  4.2  If  is  strongly  consistent  for  $  and  if  p(<I>)  <1  then 


(4.7) 


is  strongly  consistent  for  B(i),  1  =  1,2.  Hence  (4.4)  and  (4.7)  provide  strongly 
consistent  estimates  for  R  and  Q  via  (4.5)  and  (4.6)  provided  that  $  and  Q  are 
nonsingular. 


Next,  we  exhibit  the  behavior  of  the  suboptimal  filter  and  forecasts  in  the 
following  theorems  and  corollary  (cf.  Theorem  2.4,  Theorem  2.5,  and  Corollary  2.4 
of  AKKW) .  Denote  positive  (semi)-definite  by  p.(s).d. 

/N  v-v 

Theorem  4.3  If  Q  is  p.d.  and  if  strongly  consistent  estimates  of 

t,Q,R  respectively  for  which  is  p.d.  and  R^  is  p.s.d.  for  all  n^l,  then 

A  ^  Ak 

P  -*■  P  and  K  ->•  K  a.s.  as  n-*-®  where  P  and  K  are  the  estimates  of  the  steady-state 
n  n  n  n 

filter  covariance  and  gain  matrices,  respectively. 

Theorem  4.4  Let  the  hypotheses  of  Theorem  4.3  be  satisfied.  If  in  addition. 


P  ($).  <  1 ,  then 


t 
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Corollary  4.1  If  the  hypotheses  of  Theorem  4.4  are  satisfied,  then 


lim  n  1  I  x"  -  |  =  0 

n-«o  ‘'t=l'  t+k  n  t' 


a.s. 


for  any  k  > 1. 


Theorem  4.4  and  its  corollary  will  be  useful  in  establishing  the  bootstrap 
principle  which  is  discussed  in  the  next  section.  We  conclude  this  section  with 
a  statement  about  the  limiting  law  of  the  matrices  Cj^(h)  given  by  (4.2).  The 
following  theorem  follows  from  Hannan  (1970,  p.  228). 

Theorem  4.5  Let  y^  be  generated  by  (4.1).  Let  c^^ (h)  and  (h)  denote  the 
ijth  element  of  Cjj(h)  defined  in  (4.2)  and  r(h)  defined  in  (4.3),  respectively. 
Then,  for  any  Integer  H>0,  the  joint  law  of 

/n  (c^jCh)  -  l,j«l,.,.,p;  h«l,2 . H 

converges  to  chat  of  a  zero-mean  normal,  the  asymptotic  covariance  between  c^^ (m) 
and  Cj^Cn)  being 


I  { Y  j^j^(r)Yj  ^(r+n-m)  +  Yj^(r+n)Yjj^(r-m) } . 


(4.8) 


In  general,  the  covariance  of  the  c^j(h)  involves  the  forth  order  cumulants 
of  the  process  which  vanish  in  this  case  since  y^  is  Gaussian.  For  more  details 
concerning  the  asymptotic  covariances  of  c^j(h),  the  reader  is  referred  to 
Hannan  (1970)  or  Anderson  (1971). 


5.  The  Bootstrap  Principle 

In  this  section  we  justify  the  techniques  established  in  Section  3.  The 


mathematical  tools  used  in  this  section  are  those  of  Blckel  and  Freedman  (1981). 
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The  estimate  of  $  given  In  (4.4)  may  be  rewritten  as 


(j)  =  $  +  A 

n  n 


where 


(5.1) 


It  is  clear  that  the  asymptotic  distribution  of  >^($^-$)  = /n  Aj^  may  then  be 

established  via  Theorem  4.5  and  the  almost  sure  convergence  of  (4.2)  to  (4.3). 

Similarly,  in  view  of  (4.5)  and  (4.6),  the  asymptotic  distributions  of  /n(R  -R) 

n 

and  i/n(Q  -Q)  may  be  established  from  the  asymptotic  distributions  of 
/n(B^(l)-B(i]i>,  1  =  1,2,  which  in  turn  may  be  established  via  Theorem  4.5  and  the 
almost  sure  convergence  of  (4.2)  to  (4.3). 

For  example,  in  the  univariate  case  (p=l)  it  is  seen  that 


n  Var(<()^-((>) -^{7(2,2) +(J)^V(1,1) -2(j>  V(1,2)}y"^  (1) 


(5.2a) 


n  Var(B^(l)-B(l))-^{a+<{)^)^V(0,0) +44)^V(1,1) -4(J)(l-Hl)^)V(0,l)}  (5.2b) 

n  Var(B^(2)-B(2))  >  { (l+<t)^)^V(0,0)  +  4<1)\(2,2)  -  4())^(1+(J)^)V(0,2)}  (5.2c) 


where  we  have  the  set 


V(i.J)  =  Ir._<„{Y(r)Y(r+i-j) +Y(r+j)Y(r-i)}. 


Let  1^(1)  be  (5.2b)  and  Z(2)  be  (5.2c).  It  can  then  be  shown  that 

n  Var(R  -R) -►  { (1+<})”^)^5;(1)  +  (t)”^I(2)  -  2(l+())”^)())"^Ea2)}T 
n  4 

n  Var(Q^-Q)  -{E(l)  +  (l-Ht)^)^E(R)  -  2(1+(J)^)Z(R1) } 


(5. 2d) 


(5.2e) 


where  we  have  set  E(R)  to  be  (5. 2d),  n  Cov(B  (1),B  (2)) ->^1(12)  where 

n  n 

E(12)  -  {(1+(J)^)(1+(^^)V(0,0)  -2(t)^(l+<^)V(0,2)  -  2(t(l+$^)V(l,0)+4  (f^V(l,2)}, 


as  the  bootstrap  counterpart  of  C  (h),  equation  (4.2),  upon  which  all  the  esti- 

A  A  ^  A.  A 

mates  ,  Q  ,  R  ,  B  (1),  and  B  (2)  are  based, 
n  n  n  n  n 

The  bootstrap  principle  given  In  Section  3  Is  now  stated  In  the  following 
theorem. 


Theorem  5.1  Along  almost  all  sample  sequences,  as  n-»-“,  conditionally  on  the 
data,  for  all  h^O, 

(1)  C  (h)  r(h)  in  conditional  probability,  and 

n 

(2)  the  conditional  law  of  /tiCC  (h)-C  (h))  merges  with  the  unconditional  law 

n  n 

of  ^(C  (h)-r(h)). 
n 

From  Theorem  5.1  we  may  then  establish  that  the  conditional  laws  of 
*  ^ 

A  ,  »^(B  (i)-B  (1))  1»1,2  merge  with  the  unconditional  law  of  /n  A  and 
n  n  n  n 

A 

/n(B^(i)-B(i))  1  =  1,2  by  what  Is  essentially  Slutsky's  theorem. 

As  previously  suggested,  the  proof  of  Theorem  5.1  is  based  on  the  Mallows 


metrics  argument  of  Blckel  and  Freedman  (1981)  and  is  fashioned  after  Freedman 
(1984).  If  Is  a  p-dimenslonal  space  equipped  with  the  Euclidean  norm  1*1 
and  Q^l,  then  d^(p,'’)  is  the  distance  between  probability  measures  y  and  v 
in  defined  as  the  Infimum  of  ^  {| over  all  pairs  of  random  vectors 
U  with  law  y  and  V  with  law  v. 


Before  proving  Theorem  5.1  we  establish  the  following  lemmas 
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Lemma  5,1  Let  0=  ($,Q,R)  and  0  =  (4i  ,Q  ,R  )  satisfy  the  conditions  of  Theorem 
'  ■  —  n  n  n  n 

4,4.  Let  F  be  the  emperical  distribution  function  (e.d.f.)  of  the  suboptimal 
n 

''t-1  ^ 

innovations  r^  ,  t«l,...,n  generated  by  0^  and  let  be  the  e.d.f.  of  the  op¬ 
timal  innovations  r^  ^  t=>l,...,n  generated  by  0.  Then  ^  ^  almost 

surely  (a.s.)  as  n 

t-1  t-1  ^t-1  ^t-1 

Proof  Noting  that  r^  ~^t~^t  '^t  ~^t~*t  ’  view  of  Theorem  4.4,  we 


-Ir"  I  t-1  ‘'t-1 1  2 


Lemma  5.2  Let  F^  be  the  e.d.f,  of  the  optimal  innovations,  r^  ,  t*l,.,.,n 

and  let  F  be  the  common  distribution  of  r^  Then  d^(F  ,F)  -►  0  a.s.  as  n 

t  z  n 

Proof  Since  the  optimal  steady-state  innovations  are  iid  (cf.  Proposition  2.1), 

this  follows  from  Lemma  8.4  of  Bickel  and  Freedman  (1981).  □ 

Now,  let  ip  .(F)  be  the  law  of  C  (h),  h  >  0,  when  the  law  of  r^  ^  is  F. 
n,n  n  —  t 

Metrize  the  i^'s  by  d^  ^  and  the  F's  by  d^.  For  notational  convenience,  we  drop 
the  subscript  h  from  the  ij;'s  and  drop  the  superscript  t-1  from  the 


Lemma  5.3  The  are  equiuniformly  continuous  functions  of  F  on 

S  =  {F:  [  |r l^dF(r)  < c^ < w}. 


(5.4) 


Proof  The  proof  of  this  lemma  is  similar  to  Freedman  (1984,  Lemma  6.3).  Fix 
* 

F  and  F  in  S.  Construct  iid  random  vectors  t  =  l,...,n  so  that  r^  has 

4c  ik 

law  F  and  r^  has  law  F  ,  and 


dP(F,F  )  -E{lrj.-rj.r}. 


<5  *  ■  *■*»  P  '  II  ^11  ^  1.  and  k  -=  II  k||  . 

* 

It  is  clear  that  (5.6)  is  small  if  F  and  F  are  close  in  d^.  □ 

Now  let  n  .(F)  be  the  law  of  C  (h) ,  h  >  0,  when  the  law  of  r  is  F, 
n,h  n  —  t 


0 
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Metrlze  the  Q's  by  and  the  F's  by  d^.  Again,  we  drop  subscript  h  from  the 

ii's. 

Lemma  5. A  The  are  equiuniformly  continuous  functions  of  F  on  S  (cf.  5.4). 

The  proof  of  this  lemma  follows  by  bounding  (F)  (F*)  ]  in  much 

pX  p  ^ 

the  same  way  as  d,  [4'  (F) ,  4)  (F  )]  was  bounded  in  Lemma  5.3. 
in  n 

^  A  t 

For  example,  put  “  ^t^t-h“^t^t-h* 

[fin<F) .  ^  i  E{  I  (h)  I  ^ } 

The  first  part  of  (5.7)  is  bounded  as  in  (5.5)  to  (5.6).  Furthermore,  by  the 
Cauchy-Schwartz  inequality 

E{(A^(h)*A^(h)(}^  <  E{(A^(h) 1^}^ 

which  may  be  bounded  (independent  of  t)  as  in  (5.6). 

The  proof  of  part  (1)  of  Theorem  5.1  now  follows  from  Lemmas  5.1,  5.2, 

and  5.3.  The  proof  of  part  (2)  of  Theorem  5.1  similarly  follows  from  Lemmas 

5.1,  5.2,  and  5.4.  For  example,  part  (1)  follows  since  the  conditional  law 

of  C  (h)  given  the  data  differs  little  In  the  sense  of  d^^^  from  the  uncondi- 
n  1 

tional  law  of  C^(h)  by  Lemma  5.3,  because  the  e.d.f.  of  the  suboptimal  innovations, 

A 

F^,  differs  little  in  the  sense  of  d^  from  the  law  of  the  optimal  innovations, 

F,  by  the  combination  of  Lemmas  5.1  and  5.2.  This  concludes  our  proof  and  this 


J 


section 


In  this  section  we  submit  three  univariate  examples  as  emperlcal  evidence 


of  the  bootstrap's  performance  as  an  aid  In  Kalman  filtering. 

6.1  Simulation;  Normal  Errors 

In  this  example  we  generated  n  ••  100  Gaussian  observations  from  the  model 
(1.1),  (1.2)  with  and  R>lo0.  Each  parameter  estimate  was  boot¬ 

strapped  L  ■  100  tlmeso  Table  6.1.1  gives  the  sunanary  results  of  30  runs;  the 
true  asymptotic  values  are  obtained  via  (5.2)  and  are  listed  as  standard  errors. 
The  sample  standard  errors  obtained  by  Inserting  estimates  for  the  true  values 
Into  (5.2)  are  also  listed. 


TABLE  6.1.1 

A  A  A 

SE(<1))  SE(Q)  SE(R) 


Mean 

St.  Dev. 

Mean 

St.  Dev, 

Mean 

St.  Dev. 

Bootstrap 

0.106 

0.030 

1.242 

0.256 

0.779 

0.361 

Sample 

0.153 

0.068 

1.749 

0.421 

0.198 

True 

0,083 

1.269 

0.843 

Table  6.1.1:  Summary  of  the  standard  errors  of  30  runs  of  the  bootstrap  procedure 
(L-lOO)  with  n  -  100  normal  observations  from  the  model  (1.1),  (1.2)  with  ({>“0.8, 
Q-4.0,  R-1.0. 

Table  6.1.2  contains  the  summary  of  the  centered  innovations  from  the  30  runs 
described  above.  The  standard  deviation  of  the  100  innovations  as  well  as  the 
first  five  autocorrelations  are  listed.  From  (2.1)  we  may  calculate  the  true 
standard  deviation  of  the  Innovations,  /p+R  ■  2.35.  Also,  for  large  sample  sizes 
n  from  lid  data,  the  autocorrelations  have  approximate  means  and  standard  devia¬ 
tions  of  -1/n  and  l/»^,  respectively* 
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I 

I 

U 

► 

i 

i 


I 


I 


TABLE  6.1.2 


Sample 

True 

STANDARD 

DEVIATION 

2.380jp.l68 

2.350 

AUTOCORRELATION 

LAG: 

1 

-0.006jHO.213 

-O.OljD.lO 

2 

-0.007jH0.069 

-O.OljjO.lO 

3 

0.033j5).098 

-O.Oljjl.lO 

4 

-0.025j<).102 

-O.Oljjl.lO 

5 

-0.016jH0.121 

-O.Oljjl.lO 

Table  6.1.2;  Summary  (mean  +  st.dev.)  of  the  Innovations  from  30  runs  of  the 
Kalman  Filter  for  n » 100  normal  observations  from  the  model  (1.1),  (1.2)  with 
4)-0.8,  Q-4.0,  R-1.0. 

6.2  Simulation;  Contaminated  Normal  Errors 

In  this  example  we  generated  n  » 100  contaminated  normal  observations  from 

the  model  (1.1),  (1.2)  with  (|i-0.8,  Q  -  4.0(90%)  +  16.0(10%)  -  5. 2 ,  and  R- 1.0(90%^ 

+  9.0(10%)  » 1.8.  That  is,  the  state  errors  are  90%N(0,4)  + 10%N(0,16)  and  the 

observation  errors  are  90%N(0,1)  + 10%N(0,9).  The  true  asymptotic  variances  of 

the  estimates  must  now  Include  the  forth  cumulants  of  the  error  sequences  (cf. 

Hannan  (1970)  or  Anderson  (1971))  so  that  V(i,j)  of  (5.2)  is  adjusted  to 

-4  -2 

V(i,j)+<  Y  (Dy  (j)o  +K  o  6  ^6  where  <  and  <  are  the  forth  cumulants 

wx'x  w  vvi,0j,0  w  v 

of  the  state  and  observation  error  sequences,  respectively,  Y^(l)  is  Cov(x^ ,x^_^^) , 

2  2 

and  are  the  variances  Q  and  R  of  the  state  and  observation  error  sequences 
and  O’^J  0  Kronecker  deltas. 

Table  6  2,1  gives  the  summary  results  of  30  runs  in  which  each  parameter 
estimate  was  bootstrapped  L  ■  100  times.  The  true  asymptotic  standard  errors 
are  obtained  as  described  above  and  the  sample  standard  errors  are  calculated 
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by  inserting  estimates  for  the  true  values  in  (5.2). 

TABLE  6.2.1 

A  A  A 

SE((j))  SE(Q)  SE(R) 


Mean 

St. Dev. 

Mean 

St. Dev. 

Mean 

St. Dev. 

Bootstrap 

0.110 

0.024 

2.141 

0.442 

1.383 

0.437 

Sample 

0.174 

0.083 

2.793 

0.674 

1.758 

0.374 

True 

0.086 

1.848 

1.268 

Table  6.2.1:  Summary  of  the  standard  errors  of  30  runs  of  the  bootstrap  procedure 
(L  =  100)  with  n  =  100  observations  from  the  model  (1.1),  (1.2)  with  4>  =  0.8,  Q  =  5.2, 
R*1.8  and  10%  contaminated  errors. 


The  summary  of  the  centered  innovations  obtained  in  this  simulation  is 
given  in  Table  6.2.2.  In  this  case  the  true  standard  deviation  of  the  innova¬ 
tions  is  i^P+R  =«  2.74. 


TABLE  6.2.2 


Sample 

True 

STANDARD 

DEVIATION 

2.754+0.234 

2.737 

AUTOCORRELATION 

LAG:  1 

-0.014+0.019 

-0.01+0.10 

2 

-0.009+0.061 

-0.01+0.10 

3 

0.032jf^0.102 

-0.01+0.10 

4 

-0.018HW.095 

-O.OljO.lO 

5 

-0.026+0.719 

-0.01+0.10 

Table  6.2.2:  Summary  (mean  +  st.dev.)  of  the  innovations  from  30  runs  of  the 
Kalman  Filter  for  n  ■  100  observations  from  the  model  (1.1),  (1.2)  with  ({)  =  0.8, 
Q*5.2,  R-1.8  and  10%  contaminated  errors. 


4i 
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6.3  Annual  Flows  of  the  Gota  River,  Sweden 

The  following  data  are  the  annual  flows  of  the  Gota  River  near  Sjotop-Vanersburg 
for  the  period  1901-1950.  The  data  are  in  the  form  of  modular  values  (actual 
annual  flows  divided  by  the  mean)  as  given  by  Salas  et  al.  (1980). 


Years 


01-10 

0.935 

0.662 

0.950 

1.121 

0.880 

0.802 

0.856 

1.C80 

0.959 

1.345 

11-20 

1.153 

0.929 

1.158 

0.957 

0.705 

0.905 

1.000 

0.918 

0.907 

0.991 

21-30 

0.994 

0.701 

0.692 

1.086 

1.306 

0.895 

1.149 

1.297 

1.168 

1.218 

31-40 

1.209 

0.974 

0.834 

0.638 

0.991 

1.198 

1.091 

0.892 

1.020 

0.869 

41-50 

0.772 

0.606 

0.739 

0.813 

1.173 

0.916 

0.880 

0.601 

0.720 

0.955 

Model  (1.1),  (1.2)  was  fitted  to  the  data  and  the  standard  errors  of  the 
estimates  were  obtained  by  conventional  asymptotics  (sample)  and  by  the  boot¬ 
strap  with  L » 100  iterations.  The  results  are  given  in  Table  6.3.1. 

TABLE  6.3.1 
Parameters 

(p  Q  R 

1.  Estimates  0.9902  0.0273  0.0067 

2.  Standard 
Errors : 

Sample  2.5410  0.0714  0.0342 

Bootstrap  0.0916  0.0088  0.0051 

Table  6.3.1:  Summary  of  parameter  estimates  and  estimated  standard  errors, 
sample  and  bootstrap  (L = 100) ,  obtained  by  fitting  (1.1),  (1.2)  to  the  Gota 
River  data. 

We  note  that  the  standard  errors  obtained  via  conventional  asymptotics 
are  unreasonably  large  and  hence  would  be  useless  to  the  investigator  of  this 
system.  This  situation  is  probably  due  to  the  fact  that  the  signal  is  close  to 
nonstationarity  (4)  =  0.99).  The  bootstrap  however,  provides  more  reasonable  es- 
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timates  for  the  precision  of  the  parameter  estimates. 

The  bootstrap  also  allows  us  to  obtain  prediction  intervals.  We  pre- 

/s.Aer\ 

dieted  X3^  to  be  0.9029,  where  =  0.9118  is  obtained  by  filtering 

under  (j),  Q,  and  R.  The  filter  was  initialized  by  the  mean  and  the  variance 

A 

of  the  data.  The  bootstrap  estimate  of  the  prediction  error  SECx^^),  is  thus 
SE(J)  x^Q  =  0.0835. 

The  steady  state  filter  error  = E^^^(x^-x^  )  was  estimated  by  fil¬ 

tering  under  ({),  Q,  and  R  and  a  value  of  0.1847  was  obtained.  The  standard  error 
of  the  50  Innovations  r^'^  =  (y  -x^‘^)  t  =  l....,50.  was  0.202  and  the  es- 

t  tv 

timated  autocorrelation  function  for  the  first  12  lags  is:  -0.066,  -0.374, 

-0.037,  0.044,  -0.157,  0.010,  0.101,  0.073,  -0.069,  0.033,  0.057,  and  -0.066. 
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